feat(semsim): explicit category/taxon filters + search-by-profile - #1375
Open
kevinschaper wants to merge 3 commits into
Open
feat(semsim): explicit category/taxon filters + search-by-profile#1375kevinschaper wants to merge 3 commits into
kevinschaper wants to merge 3 commits into
Conversation
…roxy The similarity search API selected entities by a CURIE prefix standing in for "things of some category in some species". That only works while prefix, category and taxon coincide one-to-one, and they do not: mouse models are split across the MGI and MMRRC prefixes so no group can name them, while MGI alone covers mouse genes, genotypes and variants at once. Search now takes explicit `categories` / `taxa` / `prefixes`, which AND together. GET /semsim/filters enumerates the valid combinations from the KG rather than a hand-maintained enum, and POST /search takes a `filter` object. SemsimSearchGroup keeps working: it expands to the explicit filters it always implicitly meant. Enabling changes: - DEFAULT_ASSOCIATIONS selects by predicate rather than by association category. The old gene+disease allowlist structurally hid all 589,030 GenotypeToPhenotypicFeature edges -- every MGI and MMRRC mouse model -- plus variants and cases, with no way for a caller to opt in. Because the pool is now heterogeneous, the legacy groups must state their category to keep their old meaning; verified they still return a single category and the same top hits (MECP2/Mecp2/Rett syndrome for a Rett query). - _termset_search scores at the phenotype level instead of per (entity, phenotype). A phenotype's similarity to a query term does not depend on which entity carries it, so the old query re-expanded each phenotype's ancestors once per annotated entity: across mouse genotypes, 447,110 association rows covering 11,530 distinct MP terms, ~39x redundant. Scoring all 81,151 mouse genotypes against Rett syndrome goes from 288.7s to 2.3s (~125x). Verified equivalent, not assumed: both paths return all 81,151 entities with a maximum absolute score difference of 2.0e-15. Exact because `closure` holds no duplicate (subject, object) pairs, so count(*) and count(DISTINCT a) agree. - _flat computes closure sizes for entities the baked `closure_size` table omits. It covers genes and diseases but no genotypes, and _flat inner-joined it, so hybrid_search -- the production default -- returned an empty result set for every mouse model with no error at all. Filtering on a `nodes` table lacking category/in_taxon now raises instead of matching nothing, so a filter that cannot be honored never looks like a legitimately empty result. Same reasoning for SemsimianService, which can only express a single prefix and now refuses anything else rather than silently answering with an unfiltered superset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
…as a shim koza now bakes a closure size for every entity with a has_phenotype edge, so _flat's computed fallback is no longer what makes genotype search work. Keep it for artifacts built before that fix, where degrading to a slower correct answer still beats silently returning nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
Case entities were already reachable as search TARGETS once the association pool was selected by predicate rather than category -- filter to biolink:Case and a termset search ranks patients. The missing half is using a Case as a search SOURCE: a phenopacket has ~20 HPO terms, and no one is going to retype them to ask "which mouse models this patient". search_by_profile takes any phenotype-annotated entity id, reads its profile from the KG, and searches with an explicit target filter. Because the profile comes from the same _assoc pool for every entity type, one call expresses: Case -> Disease likely diagnosis for a patient Case -> Genotype mouse models for a patient (needs mouse taxon filter) Genotype -> Case patients resembling a mouse model (the MMRRC<->phenopacket link) Engine gets a thin `profile(entity)` accessor; the service composes it with the existing search and returns (query_profile, results) so a caller can weight each result by how many terms drove it. The endpoint is guarded to the ducksim backend -- semsimian has no KG to read a profile from -- returning 501 rather than a confusing failure, and 404 when the entity has no phenotypes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
✅ Deploy Preview for monarch-app ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1375 +/- ##
==========================================
- Coverage 83.34% 83.30% -0.05%
==========================================
Files 126 126
Lines 6648 6779 +131
==========================================
+ Hits 5541 5647 +106
- Misses 1107 1132 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Open
9 tasks
Member
Author
|
Part of #1376 (umbrella: ducksim-only similarity features). This PR delivers the explicit corpus filters, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes semantic-similarity search explicitly filterable and profile-driven, so it can
target arbitrary entity sets — in particular MMRRC mouse genotypes scored against
human disease. Depends on monarch-initiative/koza#245 (closure_size covering
genotypes) for those entities to have profile sizes in
monarch-kg.Commits
ducksim now takes explicit
category/taxon/entityfilters instead ofinferring scope from a CURIE-prefix heuristic. Adds
SemsimSearchFilter/request models and a phenotype-level termset-search rewrite. (ducksim.py,
semsim.py, additional_models.py, ducksim_service.py, semsim_service.py, + tests)
the on-the-fly closure_size compute becomes a documented fallback now that koza
bakes it into
monarch-kg.search_by_profilescores an entity (gene / disease / genotype / MMRRC strain)against the corpus using its own phenotype set. (+ unit + router tests)
Why
The prefix-proxy filter couldn't express "score these mouse genotypes against human
disease," which is exactly the MMRRC archiving use case. Explicit category/taxon/entity
filters plus profile-driven search make that a first-class query.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2